Method of defining and performing dynamic user-computer interaction, computer guided navigation, and application integration for any procedure, instructions, instructional manual, or fillable form

ABSTRACT

Various embodiments described in this disclosure relate to methods and computer-based systems that implement those methods for overlaying or superimposing computer-user interaction widgets and application interface widgets on top of content blocks contained within digitized documents which represent written procedures or instructions, instructional manuals and fillable forms, in order to support means of providing dynamic user-computer interaction and computer guided navigation, as well as application functions and data integration, driven through structured interaction and integration metadata definitions, as it relates to said content blocks, during job performance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/353,895 filed on Mar. 14, 2019, entitled “Method Of Defining AndPerforming Dynamic User-Computer Interaction, Computer GuidedNavigation, And Application Integration For Any Procedure, Instructions,Instructional Manual, Or Fillable Form,” which claims the benefit of andpriority to U.S. Provisional Patent Application Ser. No. 62/676,197filed on May 24, 2018 and entitled “Method of Defining and PerformingDynamic Human-Computer Interaction, Computer Guided Navigation, AndApplication Integration for Any Procedure, Instructions, InstructionalManual, or Fillable Form,” which applications are expressly incorporatedherein by reference in their entirety.

BACKGROUND Background and Relevant Art

Across any industry that requires a structured approach to jobperformance, many jobs performed by people are guided by writtenprocedures and work instructions, which ensures reliable and consistentperformance of tasks and minimizes the risk of user errors. Eveninstructional manuals for goods, requiring setup and maintenance,contain instructions that need to be followed by a person using suchgoods.

Both instructional manuals as well as written procedures/instructionsdirect the actions of a person performing the job in a defined sequence,while, optionally, providing other information and documents such asdiagrams, drawings, illustrations, images, and fillable forms.

Instructional manuals, written procedures and work instructions areoften presented to one who performs a job in the form of paper documentsor an equivalent digitized document, such as a PDF, on a computerdevice.

In more mission-critical industries, such as Nuclear or Oil & Gas, fieldworkers often carry a large stack of paper work packages, or anequivalent digitized set of documents on a computer device, to the jobperformance site just to complete a single work order. Often suchprocedures and instructions encompass multiple different scenarios andthus require complex navigation of documents, whether physical documentsor electronic, to the applicable scenario by the field worker.

Field workers must often manually sift through volumes of informationpertaining to irrelevant scenarios in order to navigate to theapplicable steps, diagrams and instructions. Not only does this reducethe “wrench time”, or the time spent actually performing the task onhand, but also may lead to unintentional user errors. Paper proceduresand instructions or their equivalent digitized documents providedrequire manual and time consuming navigation to relevant information andthe next step, computational support, placekeeping of instruction steps,verification of tools and equipment pertaining to theprocedure/instructions, verified acknowledgment of Cautions and Noteswithin instructions or other content blocks, verified completion ofsignatures, and support for other aspects of job performance.

Furthermore, non-interactive written procedures/instructions only allowfor job monitoring once the job is complete, and only based on anemployee's report of the job.

There are various innovative approaches in defining and performinguser-computer interaction and computer guided navigation when it comesto the performance of written procedures/instructions. However, anyapproach that requires fundamentally changing the wayprocedure/instructions are authored will suffer from significant changemanagement costs and user adoption issues. For example, a typicalnuclear utility invests tens of millions of dollars in the process ofwriting, reviewing, approving, and organizing tens of thousands of workinstructions and procedures. In a nuclear utility, procedure writerscurrently use advanced, mature and full-featured text authoring toolssuch as Microsoft Word or Adobe FrameMaker to define procedures andwork-instructions. Abruptly changing that with a new tool for authoringinstructions as well as user-computer interaction is not only a colossalundertaking but also suffers from tremendous change management costs inusing the new process and tools for procedure-writing. Besides thechange management costs, such approaches pose a significant risk ofadoption and user acceptance as currently procedure writers areaccustomed to mature software for authoring instruction (without theinteraction or application integration).

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

The scope of the present invention is not limited to any degree by thestatements within this summary.

Various embodiments of the disclosure relate, generally, to computerimplemented systems and methods to define user-computer interactions,computer guided navigation and application integration for digitizedwritten procedures/instructions, instructional manuals and/or fillableforms (hereon referred to collectively or individually as the digitizeddocument), in a structured metadata format containing the interactionand interface definitions together with the corresponding content blocksof the digitized document, and to perform the defined interactions andintegrations guided through the structured metadata at job performancetime. The computer guided interaction and application interfacedefinitions may be defined through an interaction authoring tool(Interaction Authoring Tool—IAT) by overlaying or superimposinguser-computer interaction widgets (heron referred to as interactionwidgets) and application integration widgets (hereon referred to asinterface widgets) over content blocks of the digitized document toencapsulate each content block through a corresponding widget and thenconfiguring the interaction and interface properties of said widgetincluding configuring its sequence of appearance statically orconditionally based on dynamic data, relative to other widgets, forcomputer-aided navigation at job performance time. The computer-guidedinteraction and application interface definitions (Interaction andInterface Definition Metadata—IIDM), together with content blocksencapsulated through the widgets, are stored in a persistent computerdata store as structured XML or functionally and sufficiently equivalentdata structures. Based on some embodiments of the disclosure the IIDMmay also be generated programmatically. At job performance time,computer guided interactions and application integration may beperformed within an Interaction Performance Tool (IPT) driven by theinterpretation of the IIDM.

Other embodiments of the disclosure relate, generally, to definingconfiguration driven interaction widgets which encapsulate contentblocks from the digitized document with each widget supporting specifictypes of user-computer interactions, as it relates to its correspondingcontent block, and embedding select widgets within other widgets formore granular encapsulation and context sensitive interactions.

Yet other embodiments of the disclosure relate, generally, to methodsfor generating IIDM programmatically or through IAT, capturing andaggregating information about the dynamic user-computer interactionsduring job performance to provide job performance monitoring, statisticsand analytics and supporting computer-guided navigation from thecontents encapsulated within one widget to contents encapsulated withinanother through static sequencing of widgets or dynamically through theautomatic evaluation of conditional statements based on dynamic data.Furthermore, to methods accommodating advanced computer guidance for jobrequirements such as repeated action steps and other advancedinteractions through different types of superimposed/overlaid and linkedinteraction widgets.

Yet, other embodiments of the disclosure relate, generally, to methodsof defining, and supporting at job performance time, the bidirectionalintegration of data and automatic dispatch of application functions fromother applications to/from content blocks of an existing digital writtenprocedure/instructions or fillable form, integrations which may bedefined through the selection of software service interfaces with typedinput/outputs such as standards-based Web Services (e.g., WSDL/SOAP orRESTful) that are automatically associated with a set of interfacewidgets that may be placed inside of any interaction or any widgetencapsulating said content blocks (container widget). Furthermore, someembodiments of the disclosure relate, generally, to providing integratedmethods through typed data interface widgets, to gather user inputsduring job performance as it relates to the contents of one interactionwidget and making such data referenceable by and available to otherproceeding widgets to support dynamic computer-aided navigation andreduce user errors during job performance.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims, or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, and other aspects, features, and advantages of severalembodiments of the present disclosure will be more apparent to one ofordinary skill in the art from the following Detailed Description aspresented in conjunction with the following several appended drawings,understanding that these drawings depict only typical embodiments andare not therefore to be considered to be limiting in scope, including:

FIG. 1 shows a flowchart of an embodiment of a method to defineuser-computer interaction and computer-aided navigation throughsuperimpose/overlay of interaction widgets on top of content blocks ofany digitized written procedure/instructions, instructional manual andfillable forms, resulting in structured metadata definitions.

FIG. 2 shows a flowchart of an embodiment of a method to defineapplication and data integration through superimpose/overlay ofinterface widgets on top of content blocks, or portions thereof, of anydigitized written procedure/instructions, instructional manual andfillable forms, resulting in structured metadata definitions.

FIG. 3 shows a flowchart of a method to provide/perform, at jobperformance time, user-computer interaction and computer-aidednavigation as well as application and data integration throughsuperimpose/overlay of interaction and interface widgets on top ofcontent blocks of any digitized written procedure/instructions andfillable forms based on structured metadata definitions.

FIG. 4 shows an example of an embodiment of the Interaction AuthoringTool (IAT), displaying an example of the digitized document, and anexample of an interaction widget palette, with a Section interactionwidget, selected from the palette and, superimposed on the content blockof the first section of the digitized document together with an exampleof some of the widget's properties and attributes.

FIG. 5 shows the same digitized document as FIG. 4, with moresuperimposed and linked interaction widgets, where one of theinteraction widgets contains child interaction widgets (or interactionsub-widgets) that correspond to instruction steps within the contentblocks contained by the parent widget and with more space being added,dynamically, to the displayed digitized document to accommodate spacefor the Acknowledgment related interactions of some superimposedinteraction widgets.

FIG. 6 shows the use of a Conditional Action step widget, within IAT, todefine automatic branching and navigation to the next interaction widgetbased on dynamic user input later provided at job performance timewithin the Interaction Performance Tool (IPT).

FIG. 7 shows an example of an embodiment of the Interaction PerformanceTool (IPT), performing user-computer interaction, computer-guidednavigation and placekeeping based on the superimposed/overlaidinteraction widgets and their configured attributes as partiallyshown/defined in FIG. 6.

Elements in the several figures are illustrated for simplicity andclarity and have not necessarily been drawn to scale. Also, common, butwell-understood elements that are useful or necessary in commerciallyfeasible embodiments are often not depicted in order to facilitate aless obstructed view of these various embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The efficiency of performing procedures/instructions may besignificantly improved when dynamic information about the job, workenvironment or equipment/tools is presented, in a graphical userinterface, to the person performing the job. Indeed, embodiments areable to provide information to a user that was not previously available.Some of this information resides in the core backend enterpriseapplications. Also, data entered during job performance, that may becaptured using fillable forms or segments thereof, most often belong tothe core backend enterprise applications such as Asset Management orWork Management systems. Furthermore, sometimes at job performance time,the ability to conveniently access a function of those backendapplications while performing procedures/instructions, for example beingable to create a work request through a Work Management system, maysignificantly improve job efficiency. Thus, embodiments implement animproved computing system that is able to display a standardizedprocedure/instruction manual or fillable form, while also providinginteractive access with additional enterprise data while the user isinteracting with the manual or form. Existing writtenprocedures/instructions, instructional manuals or fillable forms,whether in paper form or as a digitized document do not provide suchfunctionality. For example, existing computer systems providing suchmanuals and forms do not also include interactive instructions guiding auser through the form, nor do such systems provide access to additionalenterprise data related to the form.

By separating concerns as it relates to instruction authoring vsinteraction/interface authoring, applying a method of embodiments ofthis invention minimizes change management costs and results in anon-intrusive and incremental approach to adding computer guidedinteraction and application integration to the performance of existing,as well as new, written procedures/instructions without requiring anysignificant change to the established existing processes of anorganization and, instead, by adding a layer of processes, and computerguided interaction, on top of what already exists.

The application of a method of embodiments of this invention provides anon-intrusive, practical and incremental approach and means to improvework efficiency and reduce user error through computer guidedinteraction for navigation to the relevant job scenario, information andsteps, computational support, automatic placekeeping of instructionsteps, verification of tools and equipment pertaining theprocedure/instructions, verified acknowledgment of Cautions and Noteswithin procedures, and for other aspects of performing a job.

The following description is meant to enable one of ordinary skill inthe art to practice the disclosed embodiments by the way of example andspecific details of its key elements. The use of specific details,arrangement of components, specified features, examples or the like donot signify a preferred embodiment and are not intended to limit thescope of this invention to said details or examples. In addition, asubset of the components of this embodiment may be arranged by one ofordinary skill in the art in different configurations than specified andall drawings accompanying this disclosure are for illustrative purposesonly. Therefore, one skilled in the relevant art will recognize,however, that the embodiments of the present disclosure can be practicedwithout one or more of the specific details, or with conventionalmethods used in the industry or other methods, components, and so forth.

In general, well-known structures and operations are not shown ordescribed in detail to avoid obscuring aspects of the presentdisclosure. Other elements not described herein relate to variousconventional methods, components and acts that one with ordinary skillin the art may incorporate in accord with the embodiment of thisdisclosure. For example, details concerning exception-handling, timingand other similar considerations have been omitted where such detail isnot necessary for obtaining a complete understanding of this closure byone of ordinary skill in the art. Furthermore, any alterations andfurther modifications in the illustrated methods, and such furtherapplication of the principles of the invention as illustrated herein arecontemplated as would normally occur with one skilled in the artincorporating the method of this invention.

As noted above, the following description is not to be taken in alimiting sense, but is made merely for the purpose of describing thegeneral principles of exemplary embodiments. Many additional embodimentsof this invention are possible. It is understood that no limitation ofthe scope of the invention is thereby intended. Reference throughoutthis specification to “one embodiment,” “an embodiment,” or similarlanguage means that a particular feature, structure, or characteristicthat is described in connection with the embodiment is included in atleast one embodiment of the present disclosure. Thus, appearances of thephrases “in one embodiment,” “in an embodiment,” and similar languagethroughout this specification may, but do not necessarily, all refer tothe same embodiment. Also, the phrases “at least one,” “one or more,”and “and/or” are open-ended expressions that are both conjunctive anddisjunctive in operation.

Moreover, no requirement exists for a system or method to address eachand every problem sought to be resolved by the present disclosure, forsuch to be encompassed by the present claims. Furthermore, no element,component, or method step in the present disclosure is intended to bededicated to the public regardless of whether the element, component, ormethod step is explicitly recited in the claims. However, that variouschanges and modifications in form, composition of components and soforth may be made, without departing from the spirit and scope of thepresent disclosure, as set forth in the appended claims, as may beapparent to those of ordinary skill in the art, are also encompassed bythe present disclosure.

In one embodiment of the disclosure, generally, a digitized form of thewritten procedure/instructions, instructional manual and/or fillableforms, in PDF with a rich content format or any other electronic format,referred to as the digitized document, is imported into an InteractionAuthoring Tool (IAT) as illustrated at step 101 of FIG. 1 and displayedas illustrated at step 102.1 in the background using a scrolledgraphical user interface window on a display of a computing device. TheIAT 402 also displays, in a separate window, a rich palette ofconfigurable interaction widgets as illustrated at step 102.2 where eachwidget in the palette, generally or specifically, corresponds to adifferent content type within a typical digitized document. FIG. 1 showsa flowchart that depicts the simplified steps, in one embodiment of thisdisclosure, used to define user-computer interaction and computer-aidednavigation through superimpose/overlay of interaction widgets over thecontent blocks of the digitized document. In FIG. 1, steps 104 and 105,the person defining the interactions (the interaction designer)superimposes (to show the content underneath) or overlays (to block thecontent underneath or portions thereof) a selected widget, from saidpalette, over (or on top of) a corresponding content block and resizesthe widget to contain and snag the desired content block (depicted inFIG. 1, step 106).

For example, the interaction designer, may superimpose a Cautioninteraction widget, over the Caution content block of the underlyingprocedure/instructions, resize the widget to contain the written Cautionsection, then set the attributes of the widget such as job performancesequence designator (e.g., sequence number or other ordered designator),that this instance of Caution requires acknowledgment, where to placeand what to label (with a label for pre-acknowledgment and another forpost acknowledgment) the means of acknowledgment inside the widget, thedisplay properties of the acknowledgment, so on so forth, to indicate anacknowledgement of the Caution section is required before the user maymove to the next content block of the procedure/instructions at jobperformance time, and to set other properties available on the widgetthat may include look and feel properties. Similarly, the interactiondesigner may superimpose Step interaction widgets over each step of theprocedure/instructions, set their display attributes, and sequence,through e.g., sequence attribute numbers, and link the steps widgetsbased on the order of performance, or conditionally, through binaryquestions, or more complex conditional statements, based on differentjob scenarios and thus provide automatic, dynamic, computer guidednavigation of instructions within the Interaction Performance Tool (IPT)702 (see FIG. 7) at job performance time.

FIG. 4 shows one embodiment of IAT 402, where an example of writtenprocedure/instructions, used for the inspection of a heater treaterpump, is displayed as the digitized document 404 in a scrolled window,with a widget palette 406 displayed in a separate window in the top,right portion of FIG. 4. In this example, an interaction designer hadalready selected a general-purpose Section widget 408 from the palette406 and superimposed it over the first section of the digitized document404, the section titled “Purpose”, and had configured some of theproperties/attributes of the selected widget, displayed in a window 410which is located below the widget palette display in FIG. 4. Forexample, the selected Section widget 408 is configured to have a“Section Name” attribute with a value of “Purpose” and to be a“Collapsible” widget meaning that the widget may be displayed collapsedor expanded both within IAT 402, when defining interactions, and throughIPT 702 (see FIG. 7), when interactions are being performed. The“Layering Effect” of the selected Section widget 408 is set to“Superimpose” so the widget 408 is superimposed (being transparent) overthe “Purpose” section vs being overlaid (which would cover the contentblock under it). The selected Section widget's sequence number (underSequence->Current) is set to “1.0”, the previous widget's sequencenumber (Sequence->Previous) is not set since the selected widget is thefirst widget, and the sequence number of the next widget to be performed(Sequence->Next), at job performance time, is set to “2.0” (i.e.Sequence->Current+1) by default. The interaction designer may change theSequence->Next attribute to point to a widget with a different sequencenumber or to use a conditional widget to dynamically determine the“Next” widget to automatically navigate to at job performance time, aswill be described later. Furthermore, the selected widget is configurednot to require an acknowledgement by the user of IPT 702 at jobperformance time, before the user may mark the widget as completed andproceed to the next widget containing the next content block to beconsidered/performed. The selected Section widget, as it may be the casewith all interaction widgets, also snags the contents of the section orthe content block contained within it as part of its metadata togetherwith the value of its attributes as configured by the interactiondesigner. One with ordinary skill in the relevant art will acknowledgethat considering the embodiments described in this disclosure, there arevarious different interaction attributes and IAT 402 designs that may beconceived utilizing conventional methods of implementation available inthe industry.

As illustrated in FIG. 1, step 107, if the digitized document is of arich content format, such as PDF, the IAT 402 also captures the contentof the content block contained by the selected widget as illustrated atstep 107.1. In FIG. 1, step 109, the interaction designer configures therelevant user-computer interaction provided by the widget to bedynamically enforced when the contained content block of the digitizeddocument is being followed at job performance time.

As illustrated in FIG. 1, step 108, when a superimposed/overlaid widgetneeds to occupy a space that is larger than the space provided by thespecific area of the displayed digitized document that is beingcontained by the widget, the displayed digitized document is modified,see steps 108.1 and 108.2, to add the required space and the modifiedversion is displayed instead of the original. As an example, when aCaution widget is superimposed over a Caution section of a digitizeddocument, requiring acknowledgment at job performance time, if there isno room between the Caution section and the next section to insert anacknowledgment button, a modified version of the digitized document,containing the required added space will be created and the modifiedversion will replace the last displayed digitized document in theassociated display. One way to accomplish the modification is to splitthe digitized document into two documents where the first documentcontains all the contents up to the end of said Caution section and thesecond document contains all the content of the digitized document afterthe Caution section and then to add the desired blank space to the endof the first document and merge back the first and the second documentwhile readjusting all the coordinates of the widgets alreadysuperimposed/overlaid in locations of the second document to reflect thenew coordinates resulting from the addition of the blank space anddisplaying the merged document instead of the last displayed digitizeddocument.

Once the interaction designer has repeated the steps 104 through 110 ofFIG. 1, for each desired content block or sub-content block (i.e.content within a content block already contained within an interfacewidget), repeating to contain all the desired content/sub-content blocksof the displayed digitized document, the interaction designer moves tostep 111 to define the order of appearance/activation of the widgets atjob performance time. One of the attributes in common across all widgetsis the attribute that defines the order of appearance/activation acrosscontent types contained by the widgets, for example through a sequencenumber attribute, resulting in computer-guided navigation at jobperformance time. The order of appearance may be defined by staticallylinking/pointing one widget to another, for example by one widgetreferencing another widget's sequence number as its next widget tonavigate to, or conditionally and based on dynamically gathered data atjob performance type, one widget may point to one or more possible nextwidgets depending on dynamic evaluation of a conditional branch. Forexample, if a condition is evaluated to true, a widget that currentlycompleted its interaction may point to the sequence number of one nextwidget, and if the condition is evaluated to be false, to another nextwidget. These expressions may be expressed through special types ofinteraction widgets such as a Conditional Action widget described inthis disclosure.

FIG. 5 shows the same digitized document 404 as FIG. 4, with moresuperimposed and linked interaction widgets, where one of theinteraction widgets contains child interaction widgets, such as childinteraction widget 412.1, (or interaction sub-widgets) that signifysteps within the parent widget and with more space being added to thedisplayed digitized document to accommodate space for the Acknowledgmentrelated interactions needed for some of the superimposed interactionwidgets configured to require acknowledgment. Note that the metadata foran interaction sub-widget, may not be required to contain the snaggedimage or the content contained within said sub-widget, since the parentwidget already contains said content as part of its metadata and theposition of sub-widget may be stored relative to the position of itscontaining parent widget.

As shown in FIG. 5, the “Purpose” section is encapsulated by aninterface widget 408 as also shown by FIG. 4 and explained earlier.Explaining further, the Section interface widget 414 associated with thecontent block in the section titled “Scope” was configured to requireacknowledgment of consideration/performance at job performance time, andsince there was not enough space to insert an “Acknowledge” button (as ameans for enforcing the acknowledgment at job performance time), inbetween the sections titled “Scope” and “Responsibilities” in theoriginal displayed digitized document 404 within the Section interfacewidget 414 associated to the “Scope”, more space was added (refer toFIG. 1, step 108 and the explanation of the step provided earlier).Furthermore, the sequence numbers (signified by the Sequence->Currentand attribute shown in FIG. 4) associated with the interface widgets, inthis case, corresponds to the numbering of the sections/sub-sections ofthe digitized document. Accordingly, in this case, the Previous and Nextwidget references for each widget corresponds to the previous and nextsection's numbering. Furthermore, as demonstrated in FIG. 5, theinteraction Section widget 412 associated with the fourth section of thedisplayed digitized document, titled “PRECAUTIONS & LIMITATIONS”,contains interaction sub-widgets corresponding to each step of saidsection. These sub-widgets have hierarchical sequence numbers such as“4.1”, “4.2”, assigned based on the sequence number of their parentwidget.

Once the interaction designer has contained all the desired content (orsub-content) blocks through corresponding interaction widgets (orsub-widgets) and finished the configuration and static or dynamic andconditional sequencing of the widgets, information associated with allthe widgets is converted to structured metadata and saved as IIDM (orother appropriate data) as illustrated in FIG. 1 steps 112 and 113. Theinformation includes, but is not limited to, the superimposed/overlaidposition of the graphical widgets with respect to the displayeddigitized document, a snagged image corresponding to the content blockof the digitized document that is contained by each widget (note thatthis may not be needed for interaction sub-widgets as noted earlier),all the original content of the corresponding content block contained bythe widget if available and needed, and all of the widget properties,conditional and sequencing information and other attributes. Also, acopy of the last displayed version of the displayed digitized documentis saved with the metadata.

Yet, in another embodiment of the disclosure, generally, a set ofinteraction widgets are provided to correspond, more specifically, tothe sections within typical procedure/instructions, including but notlimited to, Purpose, Scope, Responsibilities, Precautions andLimitations, Prerequisites, Parts and Materials, Instructions,References, and Attachments or a General Content widget that may containany content block. Alternatively, or in conjunction with suchspecialized widgets associated with each typical section or contentblock, a general-purpose Section widget (as shown in FIG. 5), with a“Section Name” as one of its configurable attributes, may be providedthat may be generically mapped to any section within the digitizeddocument by the interaction designer and its “Section Name” attributecan be set to the title of the mapped section.

For example, if the Section widget is mapped to a section of thedigitized document titled “Purpose”, then the “Section Name” attributeof the widget may be set to “Purpose” by the interaction designer (or asdiscussed later through automation). Furthermore, another set of widgets(or sub-widgets) may be pre-configured to associate to the typicalsubsections under the Instructions section, including but not limitedto, Step or Action Step, Time Dependent Step, Continuous Action Step,Note, Caution, Warning, Conditional Action, Branch (or Go to), ComponentVerification, Reference (or Refer to), Bulleted list, Figure, Table,Critical Step, and Hold/Inspection Point or Independent Verification.These typed widgets are examples of widgets that may be available in thedisplayed “Interaction Widget Palette” referred to in FIG. 1, step102.2.

In some embodiments, a general Section widget with general and furtherconfigurable interaction properties is mapped to a section of a contentblock when no explicit keyword association is configured to match with atitle of what is assumed to be a section of the content block. One ofthe attributes of the widget can capture the title, if any, of thesection.

In some embodiments, a General Container or Content widget with generaland further configurable interaction properties is mapped to a contentblock when no explicit keyword association is configured and matchedwith the title of a content block. One of the attributes of such awidget is designed to capture the title, if any, of the content block.

Yet, in another embodiment of the disclosure, generally, an interactionwidget associated with some content type may contain child interactionwidgets and those child interaction widgets may contain other childinteraction widgets to any level of depth. One approach to capturing theorder of widget navigation is to associate hierarchical sequence numbersto widgets. For example, if the interaction widget associated with anInstruction section of the digitized document has a sequence number of5, the first child Step interaction widget corresponding to the firststep of the Instruction widget will have a sequence number of 5.1, thesecond child Step widget will have a sequence number of 5.2 and so on.This scheme for sequence number can accommodate any level of widgetparent/child relationship. The content type associated with a parentinteraction widget will be marked completed automatically when all itsrelevant child interaction widgets are marked completed.

Yet, in another embodiment of the disclosure, generally, the set ofavailable widgets, their interaction and interface behavior and theirassociation with the sections and content blocks ofprocedure/instructions may be configuration-driven. One way to approachthe configuration mechanism is to introduce a superset widget thatprovides a superset of capabilities, for all possible interactions, andproperties to turn those capabilities on/off, say with defaults set tooff, and then to allow the user to define any number of named customwidgets each with a relevant subset of those capabilities, by turning oneach desired capability through configuration. For example, the user maydefine a widget named “Caution” widget, using an instance of saidsuperset widget, and only turn on the “acknowledgment required” andplacekeeping interaction capabilities of the associated instance ofsuperset widget through an XML-based configuration mechanism.

Yet, in another embodiment of the disclosure, generally, a dynamic Tableof Contents (TOC) of all the widgets superimposed/overlaid on top of thedigitized document is represented on a display, in the order defined bythe links across the widgets, through their sequence numbers, or othermeans of sequencing, and with each entry in the TOC uniquelyrepresenting a superimposed/overlaid widget, in a manner that the TOCcan be used to navigate to a particular widget in the display thatcontains the content blocks of the digitized document through hyperlinksor a similar manner both within IAT 402 and IIP.

Yet, in another embodiment of the disclosure, generally, the interactiondesigner may define named views and then associate eachsuperimposed/overlaid widget to one or more of said named views throughan attribute provided on the widget for defining such associations suchthat, at job performance time, the user of the IPT 702 is able to switchto different views, where each selected view only displays the contentof the widgets associated to said view.

Yet, in another embodiment of the disclosure, generally, the interactiondesigner can divide the digitized document into two parts: 1) FrontMatter (usually containing sections such as Purpose, Scope,Responsibilities, Precautions and Limitations, and Prerequisites) vsBody (usually containing the procedure or instructions). And thenrequire the person performing the job, at job performance time, toacknowledge the Front Matter before he/she may proceed to the Body.Alternatively, the IAT 402 may provide a widget designed to containmultiple sections of the digitized document, say Section widget, in thiscase containing all the sections of the Front Matter, and turn on theacknowledgment interaction for that widget and link it to the firstwidget of the Body.

Yet, in another embodiment of the disclosure, generally, when thedigitized document is expressed in a rich content type, by extractingand storing the original contents of the content block contained by awidget with the associated widget metadata, the procedure/instructionsand the user-computer interaction widgets associated to its sections maybe also presented with a different form factor on any device such as asmart phone or smart glasses at job performance time, withoutrelying/using the digitized document in the background, and by firstdetermining the position of the next widget with the smallest sequencenumber on the device display and then displaying the content relative tothe position of the containing widget on the display, then based onwidget sequence numbers, selecting a next widget and determining itsposition relative to the first widget and then displaying its contentrelative to the position of its containing widget, until all contentblocks are displayed relative to the position of their containingwidgets on the display. Then, placing the first widget on the display ofa computational device to signify the current content block underperformance and provide placekeeping and other interaction through thedisplay of the widgets as work progresses at job performance time.

Yet, in another embodiment of the disclosure, generally, the widgetsassociated with some select sections of the procedure/instructions maybe superimposed/overlaid on top of the saved digitized documents and theperformance of some other select sections, including but not limited tosteps within select instructions, maybe transferred to another device,such as a smart glass, without relying/using the digitized document inthe background.

Yet, in one embodiment of this disclosure, generally, a widgetassociated with a Conditional Action may support branching based on anIF/WHEN [condition], in order to link the widget to another specificwidget thus causing automatic navigation to the specific widget at jobperformance time. The condition may be expressed as a binary or a seriesof binary questions, for example with Yes/No answers, linked throughlogical operators such as AND/OR/NOT or EITHER/OR, or any number oflogical operators and conditions. A Conditional Action Widget mayreference data that was previously entered by a user to make adetermination of whether a widget and its content should be displayed atall when a prior completed widget pointing to it is marked as completedor Not Applicable (N/A).

FIG. 6 shows an example of the use of a Conditional Action step widget416.1.3, within one embodiment of IAT 402, to define automatic branchingand navigation to the next interaction widget based on dynamic userinput later provided at job performance time within the IPT 702. Asshown, the interaction widget 416 associated with the “INSTRUCTIONS”section of the displayed digitized document, contains a Step interactionsub-widget 416.1 associated to the first step of the instructionssection. The Step interaction sub-widget 416.1, itself, contains fiveinteraction Step sub-widgets 416.1.1, 416.1.2, 416.1.3, 416.1.4, and416.1.5. The sub-widget 416.1.3 associated to the third sub-step (withsequence number 6.1.3) is a Conditional Action step interaction widgetoverlaid on top of the corresponding content block of the displayeddigitized document. The original instruction in the digitized documentread “If Pump cover is not installed, Go To Section 6.2”, however,because the Conditional Action widget was overlaid (vs superimposed),the original content block is covered by the widget, and the interactiondesigner configures the Conditional Action widget to read: “Is Pumpcover installed?”, and provides two buttons corresponding to twopossible binary answers of “Yes” or “No” and in this case configures the“Yes” answer to go to the widget 416.1.4 with the next sequence numberand “No” to branch to a widget 416.2 with sequence number 6.2. At jobperformance time, as shown in FIG. 7 and explained later in thisdisclosure, if the user presses the button label with “Yes”, the IPT 702displays/activates the widget 416.1.4 associated with the next sequencenumber, 6.1.4, and if the “No” is selected, the IPT 702 branches anddisplays/activates the widget 416.2 with 6.2 (and, in this case, alsoits first child widget 416.2.1 at sequence number 6.2.1.

Yet, in one embodiment of this disclosure, generally, a widgetassociated to a Repeat section is provided to support two basic forms ofrepeating: 1) by embedding a Conditional Action widget that causesautomatic navigation through branching to a prior step/widget, 2) bybeing configured to repeat a specified number of times. In any case, atjob performance time, the user will be able to see the originalperformance results and its associated data for each step withoutobscuring the use of the steps for the repeat, for example, throughmultiple tabs, where each tab is associated to an iteration of therepeat with the current iteration in the first tab. When a Repeatsection is complete, IPT 702 either navigates the user back to theorigin or to the next sequential widget, depending on the rulesconfigured through IAT 402 or the structured metadata.

Yet, in another embodiment of the disclosure, generally, an interactionwidget, for example one associated with a simple Action Step, may beconfigured to enforce time requirements. These requirements may include,but are not limited to, minimum time, maximum time, or a time range forthe performance of the widget. At job performance time, the user isprovided with a clock/stopwatch that he/she may manuallystart/stop/reset, where the clock changes visual state when the timerequirement is met, which then the user may mark the step as completed.

Yet, in one embodiment of this disclosure, generally, an interactionwidget, for example one associated with a simple Action Step, may beconfigured to require and enforce Concurrent Verification (CV), wheretwo users performing a step/section must be physically present together.In this case, both workers will perform the step together and sign off,through initials or digital signature sub-widgets, after the performanceof the step/section is completed and before they are allowed to proceedto the next step/section of the procedure/instructions. IPT 702 mayvisually render a CV step/section distinctly to ensure that the mainuser and the concurrent verifier are both present before performing thestep. Similarly, an interaction widget may be configured to require andenforce Independent Verification (IV), where the only difference is thatthe verifier does not have to be present at the time the step/section isbeing performed. When an interaction widget configured as IV is markedas complete, the IPT 702 will not yet progress to the next step, insteadit sends a notification, maybe through an embedded Notificationsub-widget, to the independent verifier with a link to the IV widgetrequiring verification, and only then will IPT 702 navigate the user tothe widget associated with the next step/section.

Yet, another embodiment of the disclosure relates, generally, to IAT 402providing procedure/instructions authoring capabilities through apalette 406 of interaction widgets used for authoring newprocedures/instructions, containing widgets types corresponding to thewidgets types used for superimposing/overlaying on top of the existingwritten procedure/instructions, except that these procedure/instructionsauthoring widgets provide means for authoring and creating new contentblocks. Widget instances from this palette may be connected andcombined, through sequence numbering, with the widgets used forsuperimposing interaction on top of existing content blocks.

Yet, in one embodiment of this disclosure, generally, any interactionwidget may be added to a set of reusable widget templates, wherereusable widgets templates are grouped by their type, for examplereusable widget templates of type Caution, and made available within theIAT 402 so they may be combined with other interaction widgets. When awidget is made reusable, the sequence number of the corresponding widgettemplate becomes dynamic and, in some such embodiments, it is onlyassigned when an instance of the widget template is connected orembedded in other interaction widgets based on the sequence number ofthe connected or containing widget and if the widget template containschild widgets, the hierarchical sequence number of the child widgets areadapted to the parent widget.

Yet, another embodiment of the disclosure relates, generally, to thebidirectional integration of data and automatic dispatch of applicationfunctions from other applications, such as enterprise asset or workmanagement systems, to/from an existing written procedure/instructions,instructional manual or fillable form, in PDF or any digitized binaryformat (the digitized document), driven by the selection of a softwareservice interface with typed input/outputs, including but not limitedto, standards-based Web Services, whereby selecting an input or outputof a service operation, based on its type, an interface widget,configured to be associated to that type, is provided within the IAT 402that may be overlaid or superimposed on any part of the content block ofthe digitized document, where said any part of content block may beencapsulated within an existing interaction widget. IAT 402 saves theinterface widget together with information regarding the selectedsoftware service and its input/outputs as structured metadata in IIDM.At job performance time, when an interface widget associated with theinput of a service is activated, the user will be prompted to providethe input within the area covered by said input interface widget andonce all inputs to a service are provided, the service will be eitherinvoked automatically, or its execution may be tied to a user triggerevent such as the press of a button through a button widget. When awidget associated with a service output is activated, at job performancetime, the area of the content block covered by the interface widgetassociated with that output will be automatically populated with theassociated output data from the execution of the service. In the casewhich both input(s) and output(s) of a service are associated withcontent blocks through multiple interface widgets, the service may beexecuted, only after all input data is entered by the user. If aninterface widget is contained within an interaction widget, it willactivate only when the containing interaction widget displays.

There are a number of differences between this method of integration andother prior art methods. One major difference between this method andother previous methods is that the previous methods either allow theuser to place an input data widget anywhere on the display or to relatethe input data widget to a fixed, inflexible location of an underlyingdigitized document. Instead, in this method, an input widget, used totake user input at performance time, or an output widget, used topopulate a portion of the display with data from other systems, isplaced relative to the position of a containing interaction widget, orany container widget used to encapsulate and contain a content block ofthe digitized document, and regardless of the digitized document andwithout requiring the digitized document at performance time. Anothernotable difference is that the inputs and outputs of a software service,such as a standards-based Web Service (e.g., WSDL/SOAP or REST), areimplicitly associated to input/output widgets at design time.Furthermore, at performance time the associated software services areautomatically executed resulting in direct integration with othersoftware applications through their business logic layer, as opposed tosimply saving the data in a database only to be integrated later. Themethod of containing the input/output interface widgets within otherlinked interaction/container widgets supports new capabilities that notonly does not require the inflexible association of the input/outputwidget to a hardcoded location of the digitized document, but alsoprovides means for interactive computer-guided navigation at jobperformance time.

Yet, another embodiment of the disclosure relates, generally, to a DataInterface Widget for simple data entry by the user, without being tiedto the inputs/outputs of a software service. The Data Interface widgetmay be superimposed/overlaid on any part of a content block that isalready contained within any other widget and the data entered throughsuch widget is made available throughout the interactiveinstruction/procedure/form to be referenced by all widgets with sequencenumbers greater than the sequence number of the Data Interface Widget orits containing widget. Furthermore, the users of IAT 402 and IPT 702 mayenter constants in interface widgets to support computation (ex min/maxvalues). The location of a Data Interface Widget is saved relative toits containing widget just as with other interaction or interfacesub-widgets.

FIG. 2 illustrates a flowchart for one embodiment of the method fordefining application and data integration through superimpose/overlay ofinterface widgets on top of content blocks, or portions thereof, ofwritten procedure/instructions, instructional manuals and/or fillableforms, resulting in structured metadata definitions saved as IIDM. FIG.2 illustrates obtaining a digitized document at step 201 and displayingthe digitized document at step 202. While FIG. 2, step 203, refers to ageneral “container widget” that encapsulates and contains (through asnagged image or capturing the contents of) a content block (step 205)of the displayed digitized document such that it can be used independentfrom the digitized document at job performance time, an “interfacewidget” described in earlier embodiments of this disclosure qualifies asa container widget and thus the process described in FIG. 1 and FIG. 2may be combined to address both requirements for user-computerinteraction and application integration in a single process. FIG. 2,step 206 accommodates creating interface widgets both based on softwareservices (see steps 206.1, 206.2, and 206.3) as described earlier orcreating or selecting interface widgets (see step 207), referred to assimple Data Interface widgets, that do not require mapping toinputs/outputs of a software service but are meant to capture dynamicuser input data, during job performance, and make such data available toother widgets for reference, for example, in the evaluation of dynamicconditionals and for dynamically branching to the next widget.Regardless of the type of interface widget, in FIG. 2 steps 208 and 209,the interaction designer superimposes/overlays the interface widget overan area of the content block contained within a container/interactionwidget and the relative position of the interface widget with respect tothe container/interaction widget is added to the widget metadata.

As illustrated in FIG. 2, step 210, if the interface widget needs tooccupy a space that is larger than the space provided by the specificarea of the displayed digitized document that is being contained by itscontaining container/interface widget, the digitized document may bemodified (see step 210.1 and 210.2) to add more space following aprocess as explained earlier for FIG. 1, step 108. Attributes of aninterface widget are configured at step 211. FIG. 2, step 212illustrates how a container/interaction widget may contain one or morechild/sub widgets. One can modify the process illustrated in FIG. 2 toallow interface widgets to contain other interface widgets or to allowcontainer/interaction widgets to contain other container/interactionwidgets. As illustrated at step 213, portions of the process can berepeated for other content blocks. The process illustrated in FIG. 2 isconcluded in steps 214, 215 and 216 where widgets are sequenced, and allwidget information and attributes are translated into IIDM and saved.

Yet, in one embodiment of this disclosure, generally, a configurableNotification interface widget is provided to cause the automaticpublication of a notification to the person performing the job and/or toother users and applications through the dispatch of a software serviceassociated to said widget. The Notification widget may be configured toevaluate dynamic input/output data. When another person is notified, thenotification message provides a link such that the person may launch aninstance of the interactive procedure/instructions/form in an instanceof the IPT 702 and automatically navigate to the same place where thenotification was originated.

Yet, in one embodiment of this disclosure, generally, a configurableEquipment interface widget is provided to integrate events and data fromspecific Internet of Things (IoT) enabled equipment or to cause suchequipment to perform a specific action at job performance time. TheEquipment widget can be configured to connect to a specific equipmentthrough its unique identifier, which may be a URL, in order to invokespecific API's on the processor of that equipment.

One of ordinary skill in the related art, may easily combine theprocesses of FIG. 1 and FIG. 2 to create a single IIDM metadatacontaining definitions of both interface widgets and interaction widgetsto provide interactions and well as integrations at job performance timefor a single digitized document that represents writtenprocedure/instructions, instructional manuals and/or fillable forms.This may be accomplished, through one embodiment of this disclosure,generally, using interaction widgets defined through the processillustrated in FIG. 1 as “container widgets” referred to in Step 203 ofFIG. 2. This can be simply accomplished by replacing Steps 201, 202,203, 204 and 205 in FIG. 2, with Steps 101 through 110 of FIG. 1. Suchthat, once all interface widgets are defined, interaction widgets may beadded inside of the interface widgets. There are other ways of combiningthe processes illustrated in FIG. 1 & FIG. 2. For example, by insertingsteps 206 through 212 of FIG. 2, in between steps 109 and 110 of FIG. 1such that the creation of interface widgets and interaction widgets maybe intermingled. Furthermore, one can create variations of the steps inFIG. 1 and/or FIG. 2, or any flowchart that is derived by combining theprocesses in FIG. 1 and FIG. 2 into a single process resulting in asingle IIDM that can be applied to the digitized document. For example,one variation may allow some interface widgets to be placed inside ofother interface widgets or their sub-widgets.

In one or more embodiment of this disclosure, at job performance time,the IPT 702 interprets the saved IIDM metadata resulting from theprocess in FIG. 1 and/or FIG. 2 or any new process resulting fromcombining the two processes, as mentioned earlier, and applying thecombination to the digitized document. In one embodiment, the IPT 702may display the content blocks encapsulated by interaction/containerwidgets through any of the following means, or other appropriatemeans: 1) by displaying a copy of the saved version of the digitizeddocument in a scrolled window and providing placekeeping andinteractions/integrations by superimposing/overlaying the firstinteraction/container widget corresponding to the first containedcontent block of the digitized document, based on the associatedcoordinates, and sequencing and other information saved in the metadata,and then when current widget is marked as completed, displaying the nextinteraction/container widget in the sequence and so on and so forthuntil the last widget is displayed and its interactions are performed.2) By displaying all the snagged images of the content blocks associatedwith each widget in the sequence the widgets are linked and relative tothe simulated placement of the first widget (without the display of thewidget itself) on an empty computer display/window and then thesimulated placement of other widgets relative to the first widget andbased on their sequence number, without displaying or relying on thesaved digitized document, and then superimposing the first widget overthe already displayed content block contained by the first widget toactivate the widget and signify the initial content block underperformance, and when said initial content block is marked as complete,to display/activate the next relevant widget to signify the next contentblock to be performed, and repeat the process until all relevant widgetsare displayed and marked as completed. 3) Similar to 2 above but,instead of displaying the image, to display the original contentcontained by the widget optionally in different form factors appropriatefor the display on different computing devices. Regardless of the meansof display, at job performance time, the widgets displayed based on thestructured metadata are used to provide computer-guidance andapplication integration and to dynamically advance the user to the nextsection of the procedure/instructions relevant to the job.

Yet, in another embodiment of this disclosure, FIG. 3 relates to aflowchart of a method to perform, user-computer interaction andcomputer-aided navigation as well as application and data integration atjob performance time, based on the structured metadata IIDM resultingfrom combining the processes described in FIG. 1 and FIG. 2 and applyingthe combined process to the digitized document. In step 301 of FIG. 3,said resulting IIDM is imported and loaded into computer memory intodata structures (that may include hash maps, linked lists, arrays or acombination of other conventional data structures) that are optimizedfor the implementation of the process in FIG. 3. FIG. 3, step 302,relates to displaying all the content blocks contained by all theinterface/container widgets, in the order of their sequence number,starting from the smallest sequence number using either of the meansnumbered “2)” or “3)” for displaying content blocks encapsulated byinteraction/container widgets, described in the above paragraph withoutrequiring a copy of the digitized document. After all the content blocksare displayed in step 302, moving to step 303 of FIG. 3, the widgetassociated to the first displayed content block is activated/displayedaround said content block and marked as the “Current widget” (see step304) to provide a placekeeping function, process all it sub-widgets (seesteps 305, 306, 306.1, 306.2, 306.3, 306.4, 306.5, 306.6, 306.7, 306.8,306.9) if any, and also to enforce its defined interactions. Once thefirst content block is considered/performed by the user and its enforcedinteractions completed, the widget may be marked as completedautomatically (see step 307), for example, upon interaction requiringacknowledgment of the content block by the user, or manually by the userwhen automatically marking as completed is not an option. After thecompletion of the first widget, which includes the processing of itsrelevant sub-widgets described in step 306 and the sub-steps thereof ofFIG. 3, if any, IIP determines the next interface widget to beactivated/displayed, if any, and marks the widget as “Current widget”(see steps 308, 309, 310, and 311).

If the first or any subsequent widget that was marked as the “Currentwidget”, through performing step 304 of FIG. 3, does not have anychild/sub-widgets relevant to the job scenario being performed asdetermined in step 305 of FIG. 3, where such determination may be madebased on conditionally linked widgets that may result in skipping theperformance of all or some of its sub-widgets, if any, by branching toanother non-sequential widget, said “Current widget” completes all itsinteractions and is marked as completed through step 307, when saidwidget points to a next interaction widget (again such next widget maybe determined conditionally), as determined in step 308 of FIG. 3, thenext widget is displayed and activated, according to step 311 of FIG. 3,and the processing of that widget will loop back to step 304 of FIG. 3thus said next interaction widget will be marked as the “Currentwidget”.

If the first or any subsequent widget that was marked as the “Currentwidget”, through performing step 304 of FIG. 3, does have one or morerelevant child/sub-widgets, where such relevance may be determinedthrough dynamic conditional branching based on job data gathered so farthrough the process, as determined in step 305 of FIG. 3, all thesub-widgets of the “Current widget” and, in the case where anysub-widget is yet another interaction widget, all of its sub-widgets,and so on and so forth will be processed by the embodiment of IPT 702according to the step 306 of FIG. 3 together with its sub-steps. One canmodify the process described in FIG. 3, to allow processing of interfacewidgets that contain other interface widgets, to any level of depth.

In step 306.1 according to one embodiment of the disclosure illustratedin FIG. 3, each sub-widget of the “Current widget” is either aninterface widget or an interaction widget. Each relevant sub-widget ofthe “Current widget”, is processed through the sub-steps of step 306.For each relevant next sub-widget being processed, when the sub-widgetis not an interface widget, as determined in step 306.1, it is impliedthat the sub-widget is an interaction widget, in which case, theprocessing continues through step 309, to step 310 of FIG. 3, where the“Current widget” that is the parent interaction widget of the sub-widgetis placed on a Stack data structure so its processing continues afterits sub-widget is processed. Then, through step 311, the interactionsub-widget is activated/displayed and processing loops back to step 304,where the sub-widget becomes the “Current widget”. Using this process,interaction sub-widgets, within other interaction sub-widgets, and so onand so forth to any level of depth may be processed. The parent widgetof an interaction sub-widget is pushed on a processing “Stack”, suchthat it can be ‘popped’ after its interaction sub-widget completes itsprocessing through step 307.c of FIG. 3, to continue its processing asthe “Current widget”. The use of a Stack data structure accommodatesembedded interaction sub-widgets to any level of depth. To keep theprocess flowchart in FIG. 3 easy to understand, many conventionaldetails associated with error-handling or special cases have beenomitted. One with ordinary skill in the art will acknowledge that suchdetail can be addressed through conventional data structures andprocesses common in the relevant art.

Referring once again to FIG. 3, when a relevant sub-widget is aninterface widget, as determined in step 306.1, if the definition of thatsub-widget is not tied to a software service, as determined in step306.2, the sub-widget is a Data Interface widget that based on itsdefined type is used to either gathered data by prompting the user orused for displaying and will be processed through step 306.6. If thesub-widget is tied to a software service, as determined in step 306.2,the processing of the interface sub-widget moves to step 306.3 where thewidget is determined to be either tied to the input(s) or output(s) of asoftware service. If it is tied to the output(s), through step 306.4,the associated software service is executed, if it was not alreadyexecuted, and the associated service output data is displayed throughthe sub-widget within the area of the content block occupied by thesub-widget. Otherwise, if the sub-widget is tied to input(s) of theassociated service, in step 306.5, the user is prompted for thereferenced input data through the associated sub-widget and the serviceis executed when the last input data is gathered through an interfacesub-widget. All the steps 306.4, 306.5, and 306.6 will go through step306.7 of FIG. 3, where all input and/or output data gathered will bemade available to be referenced and accessed by all proceeding widgetsand sub-widgets and processing moves to step 306.8, where the processingcontinues to the next sub-widget, if any, through step 306, and whereall sub-widgets of the “Current widget” are processed, processing moveto step 307 where all remaining interactions of the “Current widget” arecompleted and the widget is marked as complete and if there are anyparent widgets on the Stack, each parent widget is ‘popped’ in order andits processing is resumed as the “Current Widget”. The processillustrated in FIG. 3, does not address processing of interface widgetsas sub-widgets of other interface widgets. One with ordinary skill inthe art acknowledges that this process may be easily modified toaccommodate such processing.

If the first or any subsequent widget that was marked as the “Currentwidget” performing its interactions does not have any child/sub-widgetsas determined in step 305 of FIG. 3, and it does not point to a nextinteraction widget, as determined in step 308 of FIG. 3, there are nomore widgets to be processed and thus all relevant interactions andintegrations associated with the job are already performed, whereprocessing moves to step 312 which signifies the end of processing ofIIDM by this embodiment of IPT 702 according to FIG. 3.

FIG. 7 shows an example of an embodiment of the Interaction PerformanceTool (IPT) 702, performing user-computer interaction, computer-guidednavigation and placekeeping as shown (and partially defined) in FIG. 6.As shown in FIG. 7, the performance of steps 6.1 and 6.2 were alreadyconfirmed by the user through their associated Action Step interactionwidgets and thus they are marked as “Completed” by the IPT. Furthermore,the user has selected the button labeled “No” when the overlaid widget416.1.3 associated with step 6.1.3 was activated/displayed, andtherefore, the widgets 416.1.4 and 416.1.5 associated with steps 6.1.4and 6.1.5 were automatically marked as “N/A” and the configured textsupplied for the “N/A” state of the corresponding widgets and the widget416.2.1 with sequence number 6.2.1, corresponding to section 6.2.1 ofthe document is automatically activated/displayed and navigated to, asthe next relevant step that needs performance and thus performed by theuser.

Yet, in another embodiment of the disclosure, generally, any of thewidgets, regardless of type, provide a set of general interactioncapabilities at job performance time through IPT 702 some of which maybe configured through the IAT 402 or the structured metadata associatedwith interaction widgets and their sub-widgets. These capabilities mayinclude, but are not limited to: a) the ability to add comments to eachwidget, through say a Comment sub-widget that can be placed andrelocated to any area within said interaction widget by the user, b) theability for a user to provide feedback/change-request on a content-blockassociated with an interaction widget, for example through a Feedbacksub-widget similar to the Comment sub-widget, such that theprocedure/instructions writer may consider the feedback at a later time,c) the ability to flag any interaction widget to be added/removedto/from bookmarks for easy navigation, d) the ability to expand orcollapse all or some of the interaction widgets, including their childinteraction widgets, e) the ability to capture media (ex. video,pictures, sound) through Media sub-widgets that can be associated withany interaction widget, f) the ability for the user to overwrite thesequence in which the computer navigation is designed to follow based onthe sequence numbers associated with the interaction widgets orconditional branching, under certain circumstances, however requiringjustification in the form of enforced Comment/Feedback and initials ofthe user, g) provide the ability to jump to the current step or sectionof the interactive digitized document with one click, h) provide theability for the user to see progress status of the completed stepsagainst the total required steps local to interaction widgets associatedwith an instructions widget and its child widgets as well as overallprogress with respect to the entire procedure/instructions/form. Notethat using TAT 402, the person defining the interactions through widgetscan specify what percentage or how many units of work is associated witheach widget relative to the overall work and for sub-widgets relative tothe parent widget, i) the ability to enforce the navigation of thewidgets associated with the numbered sections and steps of instruction,and the sub-widgets associated with bulleted sections in an interactiveprocedure/instructions in order, such that sequential performance isenforced and a later step/widget is not allowed to be marked complete(or N/A if not applicable) until the prerequisite step/widget is markedcomplete, or out of sequence, such that the associated steps/widgets canbe performed in any order and marked as complete (or N/A if notapplicable, based on the selected widget properties designated throughTAT 402 or expressed in the structured metadata associated with aprocedure/instructions/form, j) the ability to automatically mark aparent widget complete when all its child widgets are dispositioned ormarked as complete, k) the ability to automatically provide centralizedmenus or Table of Contents for all widgets and sub-widgets associatedwith the sections and sub-sections as well as for the references andfigures such that the user may directly navigate to any of them throughthe menus, l) the ability to support full-text search as well assearch/view by section type such as all Cautions or all the interfacewidgets used for data entry, through which search/view the user maydirectly navigate to the content of the widgets associated to the searchresults, m) the ability for the user to pause the performance of asection/step such that the associated widget timer freezes and no stepcan be dispositioned and the display becomes read only until resumed, n)ability to zoom in/out, pan and annotate figures/diagrams, o) to supportfour different states for each widget and sub-widget: Not-started,Under-performance, Completed or N/A (Not-applicable), p) to highlightCritical steps distinctly, q) to provide an indexed view for eachwidget/sub-widget type, such as Caution, Comment, Feedback, Media orother widgets and sub-widgets associated to each content-block, so itcan be reviewed separately, and provide easy navigation to theinteraction widget against which the comment was entered and thus to thecontent-block contained by that interaction widget.

Yet, another embodiment of the disclosure relates, generally, tocapturing and aggregating information about the dynamic, user-computerinteraction during job performance to provide job performancemonitoring, statistics and analytics. Examples of such data includes,but is not limited to, location of job performance through device GPScoordinates, the elapsed time between the completion of each step ofinstruction, and other dynamic data resulting from user-computerinteraction throughout job performance.

Yet, another embodiment of the disclosure relates, generally, toautomatic generation of IIDM by programmatically reading a stream ofdigitized written procedure/instructions through a computer processorand into a computer memory, for example in PDF format, and parsing andanalyzing the contents and dispatching subroutines to programmaticallysuperimpose widget metadata corresponding to typicalprocedure/instructions labels encountered where the label to widgetassociations may be configuration-driven. For example, when a Cautionsection is encountered in the stream, a rectangular Caution interfacewidget is automatically created with its dimensions corresponding to thearea needed to sufficiently contain the content type of the associatedCaution section, and a sequence number that is automatically assignedconsidering the sequence number of the last automatically generatedinterface widget then saved within the IIDM.

Yet, another embodiment of the disclosure relates, generally, to addingpalette(s) of widgets to IAT 402, designed for encapsulating commandsthat may drive one or more computerized machine(s) capable of carryingout a complex series of actions, where each command drives one or moreaction(s), and relates to a step of an instruction encapsulated with awidget from said additional palette(s). With the practice of thisembodiment, instead of a user performing one or more of the steps withinan instruction, one or more computerized machine(s) may be automaticallydriven to perform said steps(s).

Yet, another embodiment of the disclosure relates, generally, toautomatic adjustment of the structured metadata in the IIDM associatedwith an original digitized document, based on a revised version of theoriginal digitized document, by programmatically comparing the revisedversion with the original and identifying the sections deleted and addedto/from the original document and then adjusting the IIDM to delete thewidgets associated with the deleted sections and to re-compute thecoordinates of the remaining widgets in IIDM to account for the newcoordinates of the corresponding widgets.

Further, the methods illustrated above may be practiced by a computersystem including one or more processors and computer-readable media suchas computer memory. In particular, the computer memory may storecomputer-executable instructions that when executed by one or moreprocessors cause various functions to be performed, such as the actsrecited in the embodiments.

Embodiments of the present invention may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, asdiscussed in greater detail below. Embodiments within the scope of thepresent invention also include physical and other computer-readablemedia for carrying or storing computer-executable instructions and/ordata structures. Such computer-readable media can be any available mediathat can be accessed by a general purpose or special purpose computersystem. Computer-readable media that store computer-executableinstructions are physical storage media. Computer-readable media thatcarry computer-executable instructions are transmission media. Thus, byway of example, and not limitation, embodiments of the invention cancomprise at least two distinctly different kinds of computer-readablemedia: physical computer-readable storage media and transmissioncomputer-readable media.

Physical computer-readable storage media includes RAM, ROM, EEPROM,CD-ROM or other optical disk storage (such as CDs, DVDs, etc.), magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above are also included within the scope of computer-readablemedia.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission computer-readablemedia to physical computer-readable storage media (or vice versa). Forexample, computer-executable instructions or data structures receivedover a network or data link can be buffered in RAM within a networkinterface module (e.g., a “NIC”), and then eventually transferred tocomputer system RAM and/or to less volatile computer-readable physicalstorage media at a computer system. Thus, computer-readable physicalstorage media can be included in computer system components that also(or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. The computer-executable instructions may be, forexample, binaries, intermediate format instructions such as assemblylanguage, or even source code. Although the subject matter has beendescribed in language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thedescribed features or acts described above. Rather, the describedfeatures and acts are disclosed as example forms of implementing theclaims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The invention may also bepracticed in distributed system environments where local and remotecomputer systems, which are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network, both perform tasks. In a distributed systemenvironment, program modules may be located in both local and remotememory storage devices.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Program-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), etc.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A method of performing multi-entity jobperformance using computer guided navigation, the method comprising: ata first instance of an interaction performance tool at a first computingsystem, obtaining a set of typed dynamic graphical user interfaceinteraction widgets ordered and organized based on one or more of thewidgets obtaining steps from a digitized document comprising a writtenprocedure, such that the widgets capture contents of content blocks ofthe digitized document; at the first instance of the interactionperformance tool at the first computing system, receiving userinteraction at one of the widgets by a first user indicating that theuser has completed a task from the written procedure; and as a result ofreceiving the user interaction at the one of the widgets by the firstuser, causing a notification widget to send a notification to a seconduser at a second computing system and to cause the second computingsystem to navigate in a second instance of the interaction performancetool to a location in the interaction performance tool that generatedthe notification to allow the second user to continue interaction withthe set of typed dynamic graphical user interface interaction widgets toperform additional tasks in the written procedure.
 2. The method ofclaim 1, further comprising, using an equipment interface widgetassociated with the typed dynamic graphical user interface interactionwidgets, causing Internet of Things enabled equipment to perform aspecific action from the written procedure.
 3. The method of claim 1,further comprising receiving feedback from the first user in a widgetassociated with the typed dynamic graphical user interface interactionwidgets with respect to a content block, and as a result, providing thefeedback back to the creator of the written procedure.
 4. The method ofclaim 1, further comprising at the first computing system capturingmedia using a widget associated with the typed dynamic graphical userinterface interaction widgets.
 5. The method of claim 1, furthercomprising capturing information about actions performed from thewritten procedure.
 6. The method of claim 5, wherein the informationcomprises location of action performance.
 7. The method of claim 5,wherein the information comprises elapsed time between completion ofsteps from the written procedure.
 8. The method of claim 1, furthercomprising receiving user input from the first user to pause performanceof a section or step of the written procedure such that an associatedwidget timer freezes.